The mechanism of existing style transfer algorithms is by minimizing a hybrid loss function to push the generated image toward high similarities in both content and style. However, this type of approach cannot guarantee visual fidelity, i.e., the generated artworks should be indistinguishable from real ones. In this paper, we devise a new style transfer framework called QuantArt for high visual-fidelity stylization. QuantArt pushes the latent representation of the generated artwork toward the centroids of the real artwork distribution with vector quantization. By fusing the quantized and continuous latent representations, QuantArt allows flexible control over the generated artworks in terms of content preservation, style similarity, and visual fidelity. Experiments on various style transfer settings show that our QuantArt framework achieves significantly higher visual fidelity compared with the existing style transfer methods.

The recent success of pre-trained 2D vision models is mostly attributable to learning from large-scale datasets. However, compared with 2D image datasets, the current pre-training data of 3D point cloud is limited. To overcome this limitation, we propose a knowledge distillation method for 3D point cloud pre-trained models to acquire knowledge directly from the 2D representation learning model, particularly the image encoder of CLIP, through concept alignment. Specifically, we introduce a cross-attention mechanism to extract concept features from 3D point cloud and compare them with the semantic information from 2D images. In this scheme, the point cloud pre-trained models learn directly from rich information contained in 2D teacher models. Extensive experiments demonstrate that the proposed knowledge distillation scheme achieves higher accuracy than the state-of-the-art 3D pre-training methods for synthetic and real-world datasets on downstream tasks, including object classification, object detection, semantic segmentation, and part segmentation.

Structure-guided image completion aims to inpaint a local region of an image according to an input guidance map from users. While such a task enables many practical applications for interactive editing, existing methods often struggle to hallucinate realistic object instances in complex natural scenes. Such a limitation is partially due to the lack of semantic-level constraints inside the hole region as well as the lack of a mechanism to enforce realistic object generation. In this work, we propose a learning paradigm that consists of semantic discriminators and object-level discriminators for improving the generation of complex semantics and objects. Specifically, the semantic discriminators leverage pretrained visual features to improve the realism of the generated visual concepts. Moreover, the object-level discriminators take aligned instances as inputs to enforce the realism of individual objects. Our proposed scheme significantly improves the generation quality and achieves state-of-the-art results on various tasks, including segmentation-guided completion, edge-guided manipulation and panoptically-guided manipulation on Places2 datasets. Furthermore, our trained model is flexible and can support multiple editing use cases, such as object insertion, replacement, removal and standard inpainting. In particular, our trained model combined with a novel automatic image completion pipeline achieves state-of-the-art results on the standard inpainting task.

In this paper, we tackle a novel federated learning (FL) problem for optimizing a family of X-risks, to which no existing FL algorithms are applicable. In particular, the objective has the form of $\mathbb E_{z\sim S_1} f(\mathbb E_{z'\sim S_2} \ell(w; z, z'))$, where two sets of data $S_1, S_2$ are distributed over multiple machines, $\ell(\cdot)$ is a pairwise loss that only depends on the prediction outputs of the input data pairs $(z, z')$, and $f(\cdot)$ is possibly a non-linear non-convex function. This problem has important applications in machine learning, e.g., AUROC maximization with a pairwise loss, and partial AUROC maximization with a compositional loss. The challenges for designing an FL algorithm lie in the non-decomposability of the objective over multiple machines and the interdependency between different machines. To address the challenges, we propose an active-passive decomposition framework that decouples the gradient's components with two types, namely active parts and passive parts, where the active parts depend on local data that are computed with the local model and the passive parts depend on other machines that are communicated/computed based on historical models and samples. Under this framework, we develop two provable FL algorithms (FeDXL) for handling linear and nonlinear $f$, respectively, based on federated averaging and merging. We develop a novel theoretical analysis to combat the latency of the passive parts and the interdependency between the local model parameters and the involved data for computing local gradient estimators. We establish both iteration and communication complexities and show that using the historical samples and models for computing the passive parts do not degrade the complexities. We conduct empirical studies of FeDXL for deep AUROC and partial AUROC maximization, and demonstrate their performance compared with several baselines.

预先训练的图像文本模型（如剪辑）已经证明了从大规模的Web收集的图像文本数据中学到的视觉表示的强大力量。鉴于学习良好的视觉特征，一些现有的作品将图像表示转移到视频域并取得良好的结果。但是，如何利用图像语言预训练的模型（例如，剪辑）进行视频培训（后培训）仍在探索。在本文中，我们研究了两个问题：1）阻碍后期剪辑的因素是什么因素，以进一步提高视频语言任务的性能？ 2）如何减轻这些因素的影响？通过一系列比较实验和分析，我们发现语言源之间的数据量表和域间隙具有很大的影响。由这些动机，我们提出了一种配备了视频代理机制的Omnisource跨模式学习方法，即剪辑，即剪辑VIP。广泛的结果表明，我们的方法可以提高视频检索的剪辑的性能。我们的模型还可以在包括MSR-VTT，DIDEMO，LSMDC和ActivityNet在内的各种数据集上实现SOTA结果。我们在https://github.com/microsoft/xpretrain/tree/main/main/main/clip-vip上发布了代码和预训练的剪辑模型。

大量网络视频的杠杆作用以及搜索的查询或周围文本（例如标题）提供了一种经济且可扩展的替代方案，可用于监督视频表示学习。然而，由于查询多义（即查询的许多可能的含义）和文本同构（即不同文本的相同句法结构），对这种弱视文的连接进行建模并不是微不足道的。在本文中，我们介绍了查询和文本之间相互校准的新设计，以增强弱监督视频表示的学习。具体而言，我们提出了双重校准网络（BCN），这些网络在新颖地融合了两个校准，以学习从文本到查询的修正案，反之亦然。从技术上讲，BCN在通过相同查询搜索的视频的所有标题上执行聚类，并将每个集群的质心作为文本原型。查询词汇直接建立在查询单词上。对文本原型/查询词汇的视频对文本/视频对话预测，然后启动文本或查询到文本校准，以估算修正案以查询或文本。我们还设计了一个选择方案来平衡两个校正。两个大规模的网络视频数据集与查询和每个视频的标题配对，新收集到弱监督视频表示的学习中，分别命名为Yovo-3M和Yovo-10m。 BCN在3M Web视频上学习的视频功能在下游任务的线性模型协议下获得了卓越的结果。更值得注意的是，BCN在较大的10m网络视频中培训，进一步的微调导致1.6％，而动力学400的TOP-1准确性获得1.8％，而在最先进的情况下，一些v2数据集的v2数据集则是1.6％。 - ART TDN和ImageNet预训练的动作网方法。源代码和数据集可在\ url {https://github.com/fuchenustc/bcn}上获得。

作为视频的独特性，运动对于开发视频理解模型至关重要。现代深度学习模型通过执行时空3D卷积来利用运动，将3D卷积分别分为空间和时间卷积，或者沿时间维度计算自我注意力。这种成功背后的隐含假设是，可以很好地汇总连续帧的特征图。然而，该假设可能并不总是对具有较大变形的地区特别存在。在本文中，我们提出了一个新的框架间注意区块的食谱，即独立框架间注意力（SIFA），它在新颖的情况下深入研究了整个框架的变形，以估计每个空间位置上的局部自我注意力。从技术上讲，SIFA通过通过两个帧之间的差来重新缩放偏移预测来重新缩放可变形设计。将每个空间位置在当前帧中作为查询，下一帧中的本地可变形邻居被视为键/值。然后，SIFA衡量查询和键之间的相似性是对加权平均时间聚集值的独立关注。我们进一步将SIFA块分别插入Convnet和Vision Transformer，以设计SIFA-NET和SIFA-TransFormer。在四个视频数据集上进行的广泛实验表明，SIFA-NET和SIFA转换器的优越性是更强的骨架。更值得注意的是，SIFA转换器在动力学400数据集上的精度为83.1％。源代码可在\ url {https://github.com/fuchenustc/sifa}中获得。

大规模预训练的语言模型的出现为自然语言处理的最新进展做出了巨大贡献。许多最先进的语言模型首先在大型文本语料库上进行培训，然后在下游任务上进行微调。尽管它最近获得了成功和广泛的采用，但对预训练的语言模型的微调通常会遭受过度拟合，这会导致由于模型的复杂性极高的复杂性和下游任务的有限培训样本而导致的普遍性差。为了解决这个问题，我们提出了一个新颖有效的微调框架，称为Layerwise噪声稳定性正则化（LNSR）。具体而言，我们建议注入标准的高斯噪声或势内噪声，并将微调模型的隐藏表示形式定向。我们首先提供理论分析以支持我们方法的功效。然后，我们证明了所提出的方法的优势，而不是其他最先进的算法，包括L2-SP，MixOut和Smart。尽管这些先前的作品仅验证其方法对相对简单的文本分类任务的有效性，但我们还验证了方法对问题答案任务的有效性，而目标问题更加困难，并且可以使用更多的培训示例。此外，广泛的实验结果表明，所提出的算法不仅可以提高语言模型的内域性能，而且还可以改善域外数据的域概括性能。

最近的图像入介方法取得了长足的进步，但在处理复杂图像中的大孔时，通常很难产生合理的图像结构。这部分是由于缺乏有效的网络结构可以捕获图像的远程依赖性和高级语义。我们提出了级联调制GAN（CM-GAN），这是一种新的网络设计，由编码器组成，该设计由带有傅立叶卷积块的编码器组成，该块从带有孔的输入图像中提取多尺度特征表示，并带有带有新型级联全球空间调制的双流式解码器在每个比例尺上块。在每个解码器块中，首先应用全局调制以执行粗糙和语义感知的结构合成，然后进行空间调制以进一步以空间自适应的方式调整特征图。此外，我们设计了一种对象感知的培训方案，以防止网络在孔内部幻觉，从而满足实际情况下对象删除任务的需求。进行了广泛的实验，以表明我们的方法在定量和定性评估中都显着优于现有方法。请参阅项目页面：\ url {https://github.com/htzheng/cm-gan-inpainting}。

在视频deNoising中，相邻的框架通常提供非常有用的信息，但是需要准确的对齐方式，然后才能刺激此类信息。在这项工作中，我们提出了一个多对准网络，该网络生成多个流动建议，然后是基于注意的平均。它用于模仿非本地机制，通过平均多个观测来抑制噪声。我们的方法可以应用于基于流量估计的各种最新模型。大规模视频数据集上的实验表明，我们的方法通过0.2DB提高了Denoisis基线模型，并通过模型蒸馏进一步将参数降低了47％。代码可在https://github.com/indigopurple/manet上找到。